Modeling Relative Effectiveness to Leverage Multiple Ranking Algorithms

In this work, we focus on modeling relative effectiveness of result sets to leverage multiple ranking algorithms. We use a relative effectiveness estimation technique (ReEff) that directly predicts the difference in effectiveness between a baseline ranking algorithm and other alternative ranking algorithms by using aggregates of ranker scores and retrieval features. Our ranker selection experiments on a large learning-to-rank data set shows that ReEff can provide substantial improvements over using a single fixed ranker – ReEff achieves more than 10% relative improvement on about 5% of the queries – and when using ranker and retrieval based features, modeling the relative effectiveness of rankers performs better than modeling their effectiveness independently. Further, compared to fusion, ranker selection yields different types of benefits and ranker selection using ReEff can further improve fusion for three fusion techniques.